This is a breaking change. I believe this is necessary because:

1. IcebergWriter is supposed to generate DataFile that always hold a partition value according to iceberg spec.

2. The existing code store partition value in the builder directly, making builder.clone() useless:

let builder = IcebergWriterBuilder::new(partition_A);
let writer_A = builder.build();
... // write to partition A

// done with partition A and now we need to write to partition B
// this is wrong because partition value A is still stored in the builder
let writer_B = builder.clone().build() 

An alternative is to add a new method clone_with_partition() but that would also be a breaking change and it's less clean compared to build_with_partition()

I'm fine with this change, but I want a further change as following:

async fn build(&self, partition_key: Option<PartitionKey>) -> Result<Self::R>

If the builder could be reused for creating actual IcebergWriter, I want to avoid cloning.

Hi, this is not changed.

Sorry I thought I replied earlier but it seems my comment didn't go thru.

I think your advice makes sense but it would be better to fix this in a separate PR.

Changing IcebergWriterBuilder::builder to &self means we will need to change all the writer builder along the writers chain to use &self (IcebergWriterBuilder -> RollingFileWriterBuilder -> FileWriterBuilder). Otherwise, the builder would look like the following and won't help too much since we still need to clone the inner. This also can cause further confusion to users since we have different semantics for each writer builder

    async fn build(&self, partition_key: Option<PartitionKey>) -> Result<Self::R> {
        Ok(DataFileWriter {
            inner: Some(self.inner.clone().build()), // inner builder still needs to be cloned
            partition_key,
        })
    }

Please let me know your thoughts

Thanks for explaination, that makes sense to me. It would be better to create an issue to track it.

Created a tracking issue here: #1753

I'm fine with this change, but I want a further change as following:

async fn build(&self, partition_key: Option<PartitionKey>) -> Result<Self::R>

If the builder could be reused for creating actual IcebergWriter, I want to avoid cloning.